IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 52, NO. 3, MARCH 2014 


1901 


Clarifications on the “Comparison Between SMOS, 
VUA, ASCAT, and ECMWF Soil Moisture 
Products Over Four Watersheds in U.S.” 

Wolfgang Wagner, Senior Member, IEEE, Luca Brocca, Vahid Naeimi, Rolf Reichle, 

Clara Draper, Richard de Jeu, Dongryeol Ryu, Chun-Hsu Su, Andrew Western, 

Jean-Christophe Calvet, Yann H. Kerr, Fellow, IEEE, Delphine J. Leroux, Matthias Drusch, 

Thomas J. Jackson, Fellow, IEEE, Sebastian Hahn, Wouter Dorigo, and Christoph Paulik 


Abstract — In a recent paper, Leroux et al. compared three 
satellite soil moisture data sets (SMOS, AMSR-E, and ASCAT) 
and ECMWF forecast soil moisture data to in situ measure- 
ments over four watersheds located in the United States. Their 
conclusions stated that SMOS soil moisture retrievals represent 
“an improvement [in RMSE] by a factor of 2-3 compared with 
the other products” and that the ASCAT soil moisture data are 
“very noisy and unstable.” In this clarification, the analysis of 
Leroux et al. is repeated using a newer version of the ASCAT data 
and additional metrics are provided. It is shown that the ASCAT 
retrievals are skillful, although they show some unexpected be- 
havior during summer for two of the watersheds. It is also noted 
that the improvement of SMOS by a factor of 2-3 mentioned by 
Leroux et al. is driven by differences in bias and only applies 
relative to AMSR-E and the ECWMF data in the now obsolete 
version investigated by Leroux et al. 
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Index Terms — Advanced Scatterometer (ASCAT), noise char- 
acterization, soil moisture, Soil Moisture and Ocean Salinity 
(SMOS), validation. 

I. Introduction 

I N a recent study, Leroux et al. [ 1 ] validated four global soil 
moisture data sets: 1 ) retrievals derived from L-band bright- 
ness temperature observations acquired by the Soil Moisture 
and Ocean Salinity (SMOS) mission [2]; 2) retrievals derived 
from the X-band brightness temperature data of the Advanced 
Microwave Scanning Radiometer — Earth Observing System 
(AMSR-E) [3]; 3) retrievals derived from C-band backscatter 
data acquired by the Advanced Scatterometer (ASCAT) [4]; and 
4) model-based estimates produced by the European Centre for 
Medium Range Weather Forecasts (ECMWF) [5]. 

The validation was carried out over four intensive ground- 
based observing networks operated by the Agricultural Research 
Service (ARS) of the United States Department of Agriculture 
(USDA): Walnut Gulch (WG) in Arizona, Little Washita (LW) 
in Oklahoma, Little River (LR) in Georgia, and Reynolds 
Creeks (RC) in Idaho [6]. For each of these watersheds, area- 
representative soil moisture values were derived by averaging 
the measurements of several in situ sensors located at a depth 
of about 5 cm. Based on these watershed averages, Leroux et al. 
calculated the linear correlation coefficient R, the standard error 
of estimate (SEE), the bias, and the root mean square error 
(RMSE), where RMSE 2 = SEE 2 + bias 2 . 

Given that the comparison of remotely sensed and modeled 
soil moisture data with in situ measurements is an important 
component of the overall validation effort, we regard the work 
of Leroux et al. as a valuable contribution to the ongoing 
validation efforts for global soil moisture data sets. However, 
we feel that the statements regarding the skill of SMOS and 
ASCAT given in the conclusions and abstract of Leroux et al. 
can be misinterpreted and require some clarifications. The 
analysis of Leroux et al. is repeated here for the same study 
sites and time period but using a newer version of the ASCAT 
data and after converting the ASCAT data to volumetric soil 
moisture units (m 3 m 3 ) based on porosity values from a global 
data set, thereby providing additional skill metrics for ASCAT 
that were not given in [1]. Section II presents a brief discussion 
of the data sets and methods used here, Section III discusses the 
results of our analysis, and Section IV offers conclusions. 
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II. Data Sets and Methods 

A. Data Set Versions 

Like Leroux et al., we use the version of the SMOS soil 
moisture data set available at that time from the operational 
center (version 5.01) (The current operational version is 5.51). 
To avoid a negative impact of radio frequency interference 
(RFI) on the SMOS measurements, we screened the data with 
the “RFI probability” flag (must be lower than 0.10) and the 
“SM DQX” flag (must indicate a successful retrieval, i.e., must 
not be —999; according to [7], a good value for filtering is DQX 
< 0.07). For ASCAT, Leroux et al. used data processed with the 
initial near-real-time processor implemented by EUMETSAT 
[8]. Here, we use the ASCAT data set reprocessed with the latest 
version of the software developed by TU Wien (Water Retrieval 
Package, version 5.5) which benefits from updates in the instru- 
ment calibration, model parameters, and algorithms [9], [10]. 

B. Spatial Interpolation 

Leroux et al. reprojected the ASCAT, AMSR-E, and 
ECMWF soil moisture data in their respective original spatial 
resolutions onto the SMOS grid. Here, the SMOS and ASCAT 
retrievals were interpolated to points located in the center of 
the four USDA-ARS watersheds: WG (31.72° N, 110.02° W), 
LW (34.95° N, 98.10° W), LR (31.65° N, 83.61° W), and RC 
(43.15° N, 116.78° W). The interpolation was done using a 
Hamming window with a radius of 41 km to ensure that both 
satellite data sets have similar spatial resolutions. 

C. Units 

While SMOS, AMSR-E, and ECMWF data are expressed in 
terms of the volumetric soil moisture content (given in m 3 m 3 ), 
ASCAT surface soil moisture data are presented as a degree 
of saturation (as a fraction or in percent). In order to visually 
compare the ASCAT data with the in situ measurements (given 
in m ;i m 3 ), Leroux et al. rescaled the ASCAT data between the 
minimum and maximum values of the in situ data. Nonetheless, 
in their Table II, they only reported R values (and not SEE, bias, 
or RMSE-values) because “by performing this rescaling, the 
ASCAT statistical results are improved unfairly in comparison 
with the other data sets because the bias is removed.” 

An alternative objective approach is to rescale the ASCAT 
surface soil moisture data m s into a volumetric soil moisture 
content 9 using the ancillary soil porosity (ft (given in m 3 m 3 ) 
[4], [11], [12] 

9 = m s ■ 4> (1) 

where soil porosity data may, e.g., be estimated from available 
global soil maps. This approach is used here to calculate SEE, 
bias, and RMSE also for ASCAT. We estimated the porosity 
values by applying the equations of Saxton and Rawls [13] 
to the texture characteristics of the Harmonized World Soil 
Database [14], The resulting porosity values are as follows: 
0.406 m 3 m~ 3 for WG, 0.456 m 3 m 3 for LW, 0.438 m 3 m~ 3 for 
LR, and 0.456 m 3 m 3 for RC. 


D. Masking 

All soil moisture data are masked for the occurrence of 
frost (minimum daily temperature recorded at the nearest me- 
teorological station < 0° C) and snow (ERA-Interim snow 
water equivalent > 0). Precipitation (rainfall and snowfall) was 
derived from the three hourly data generated by the Global Land 
Data Assimilation System (GLDAS) [15]. 

E. Small Sample Size 

As listed in their Table II, the sample sizes used by 
Leroux et al. for calculating the error metrics ranged between 
35 and 74 data pairs because data from only ten months 
(January 1 to November 9, 2010) were used, with morning and 
evening observations evaluated separately. Furthermore, they 
required data from all five sources to be available on each date 
to ensure a fair comparison. However, given the limited time 
period of their study, a drawback of this approach is that the 
resulting sample sizes were low, resulting in large statistical 
uncertainties. 

Therefore, unlike Leroux et al. we computed the statistics us- 
ing all available SMOS and ASCAT measurements (after mask- 
ing for frost and snow) and without differentiating morning 
from evening data. For this reason, instead of using only 50-70 
measurements, we used at least 250 SMOS/ASCAT measure- 
ments to calculate the metrics at each site. For ASCAT, we also 
calculated the statistics after masking problematic time periods 
for the WG and LR sites (see next section). Additionally, we 
repeated all computations using only morning/evening passes 
with both SMOS and ASCAT data, as was done by Leroux et al. 
(not shown). This reduced the sample size by 35%-55% 
depending on the site and sensor, but the general conclusions 
drawn for the different sites and the two sensors did not change. 

III. Results and Discussion 
A. Analysis of Time Series 

Figs. 1-4 show the volumetric soil moisture data estimated 
from ASCAT, SMOS, and the in situ measurements for the 
four USDA-ARS watersheds. The figures indicate that both 
SMOS and ASCAT exhibit a higher temporal variability than 
the in situ data. This is partly due to the higher noise of the 
satellite retrievals compared with the noise of the in situ data, 
but it also has physical reasons: While the satellite estimates 
represent the soil moisture content in the top few centimeters of 
the soil, the in situ data represent the soil moisture content in 
the subsurface soil layer centered at the installation depth of the 
sensors (5 cm). Therefore, the satellite data are more sensitive to 
atmospheric forcings (precipitation, radiation, and evaporative 
demand) which, in addition to the noise, explains their higher 
temporal variability and higher dynamic range. 

First, we consider whether ASCAT exhibits a more dynamic 
and/or noisier behavior than SMOS. This could be expected 
given that the penetration depth of the C-band ASCAT data is 
shallower than that of the L-band SMOS data. However, in our 
view. Figs. 1-4, do not show the ASCAT data to be obviously 
noisier, and so, we performed a spectral analysis (not shown) to 
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JanlO FeblO MarlO AprIO MaylO JunlO JullO AuglO SepIO OctIO NovIO DecIO Janll 


Fig. 1. Comparison of (blue dotted line) SMOS and (red line) ASCAT volumetric soil moisture (m 3 m — 3 ) with (green thick line) in situ measurements over the 
WG observation network for the year 2010. The top panel shows GLDAS precipitation (in millimeters). The time periods when the satellite data were potentially 
affected by frost and/or snow cover are indicated by the shaded light-blue lines. The problematic summer period, as discussed in the text, is indicated by the gray 
horizontal bar. 
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Fig. 2. Same as for Fig. 1 but for LW. 



JanlO FeblO MarlO AprIO MaylO JunlO JullO AuglO SepIO OctIO NovIO DecIO Janll 


Fig. 3. Same as for Fig. 1 but for LR. 

determine the stochastic component of the SMOS and ASCAT 
data [16]. This analysis showed that the stochastic component 
of ASCAT is slightly larger than for SMOS over LW and RC 
and slightly lower over WG, while for LR, the results are similar 
for both satellites. In other words, our analysis did not reveal 
any evidence that ASCAT is noisier than SMOS. The higher 


noise of ASCAT noted by Leroux et al. may be due to the fact 
that they linearly interpolated the 25-km ASCAT data to the 
SMOS grid, while here, the ASCAT data were aggregated to 
roughly the SMOS spatial resolution using a Hamming window 
approach (Section II). Recall also that a newer version of the 
ASCAT data was used here. 
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Fig. 4. Same as for Fig. 1 but for RC. 


Next, we examine the seasonal behavior of the two satellite 
data sets. Overall, SMOS retrievals follow the seasonal evo- 
lution of the in situ data quite well. This is also the case for 
ASCAT retrievals at LW and RC. Over WG and LR, however, 
there are some problematic time periods when the ASCAT soil 
moisture values are consistently higher than what would be 
expected from the in situ and the SMOS data. 

Over WG, ASCAT shows somewhat elevated soil moisture 
levels in the dry period from the beginning of May to mid- 
July (Fig. 1). Such behavior has been observed in other arid 
and semi-arid regions as well and is hypothesized to be due 
to volume scattering effects from dry subsurface soil layers 
[4]. Given that to date there is no solution to this problem, 
Wagner et al. [4] recommended that under completely dry 
conditions in arid environments, SMOS, AMSR-E, or other 
passive microwave soil moisture data (which are less affected 
by this phenomenon) should be used. Additional research is 
needed, however, to better identify the conditions under which 
the passive microwave data are preferable and to provide more 
reliable guidance to users. 

Over LR, the problematic time period for ASCAT is from 
mid-June to the end of September (Fig. 3). One potential reason 
for the high ASCAT soil moisture data in summer may be that 
the parameterization of the vegetation correction term in the 
ASCAT retrieval is not optimal over this area (the seasonal 
behavior of the vegetation correction term looks reasonable but 
the magnitude may not be large enough). However, we do not 
exclude the possibility that the increased ASCAT signal during 
this period is also partly due to enhanced backscatter over wet 
soil surfaces and wetlands. As described by Jackson et al. [6], 
the LR region “is typified by broad floodplains with very poorly 
defined stream channels and gently sloping uplands.” A visual 
inspection of optical images (e.g., on Google Maps) shows that 
there are hundreds of small lakes in this area. Furthermore, as 
can be seen in Fig. 3, according to GLDAS, rainfall occurred 
frequently during this period. Additional research that inves- 
tigates, e.g., the backscatter behavior at a scale of 50-100 m 
using synthetic aperture radar data is needed to confirm or 
refute the aforementioned hypotheses. Note that SMOS also 
overestimates soil moisture slightly at LR during the same 


TABLE I 

Error Metrics of a Comparison of SMOS and ASCAT Soil 
Moisture Data With In Situ Measurements Over Four USDA 
Watersheds. For the WG and LR Watersheds, the ASCAT 
Results Are Also Shown After Masking the Problematic 
Summer Periods (See the Row Denoted by ASCAT*). R Is the 
Correlation and N Is the Number of Data Pairs. The SEE, 
the Bias, and the RMSE Are All Given in m 3 m -3 . All Results 
Are Statistically Significant With a p- Value Less Than 0.01 




R 

SEE 

Bias 

RMSE 

N 

WG 

SMOS 

0.74 

0.041 

0.018 

0.045 

339 

ASCAT 

0.50 

0.032 

-0.007 

0.033 

367 


ASCAT* 

0.64 

0.030 

-0.014 

0.033 

316 

3t 

SMOS 

0.80 

0.045 

0.019 

0.048 

307 


ASCAT 

0.75 

0.055 

0.047 

0.073 

327 


SMOS 

0.55 

0.052 

0.058 

0.078 

346 

c* 

H-l 

ASCAT 

0.21 

0.089 

0.083 

0.122 

373 


ASCAT* 

0.55 

0.068 

0.048 

0.083 

251 

u 

SMOS 

0.65 

0.048 

-0.030 

0.057 

270 

c* 

ASCAT 

0.69 

0.060 

-0.035 

0.069 

280 


period, which may similarly be related to the presence of water 
bodies or temporary standing water. 


B. Error Metrics 

The error metrics for SMOS and ASCAT from our analysis 
are summarized in Table I. For reference, this table also shows 
the ASCAT error metrics after masking the problematic time 
periods (May to mid-July for WG and mid-June to September 
for LR). 

Overall, the correlations obtained by using morning and 
evening data for many more dates are somewhat smaller than 
those reported by Leroux et al. in their Table II, likely related 
to the differences in data versions and processing (Section II). 
With the exception of the ASCAT data over LR where the 
correlation is very low in our case ( R = 0.21), we find that 
the correlations obtained for ASCAT are comparable with those 
obtained for SMOS and are within the expected statistical 
uncertainties for the sample sizes ([1, Fig. 6] provides error 
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bars). These findings confirm the conclusion of Leroux et al. 
that “all products are equivalent in terms of correlation.” 

Finally, we now analyze the SEE, bias, and RMSE metrics 
that were not provided by Leroux et al. for ASCAT. The values 
for SMOS, as listed in Table I, agree reasonably well with 
those reported by Leroux et al. in their Table II, which suggests 
that these three metrics are somewhat less sensitive than the 
correlation to the selection and preprocessing of the data. 
Comparing the SEE, bias, and RMSE values for ASCAT with 
those for SMOS, Table I shows no clear advantage of SMOS 
over ASCAT to within the expected statistical uncertainties, 
again with the exception of the LR site when the problematic 
summer period is not masked. Over the WG site, the SEE, 
bias, and RMSE values for ASCAT are better than for SMOS, 
independent of whether the problematic time period for ASCAT 
from May to mid-July is masked or not. 

Moreover, our Table I and Table II of Leroux et al. suggest 
that the differences in the RMSE values across the data sets are 
largely driven by differences in the bias, while the SEE values 
are roughly of the same order of magnitude. The statement 
in the abstract of Leroux et al. that SMOS “represents an 
[RMSE] improvement by a factor of 2-3 compared with the 
other products” must therefore be interpreted accordingly. 

IV. Conclusion 

In this clarification on the study by Leroux et al. [1], we have 
demonstrated how the ASCAT data can be assessed quantita- 
tively in the same manner as the SMOS data by using ancil- 
lary porosity data to convert the ASCAT degree of saturation 
index into volumetric soil moisture units. Our results show that 
ASCAT data are skillful and that there is no indication that the 
ASCAT soil moisture data are “very noisy” (as stated in the 
Abstract and Conclusions of Leroux et al.). While the ASCAT 
data clearly exhibit some unexpected behavior over the semi- 
arid WG site during extremely dry conditions and over the hu- 
mid LR site during the wet summer period, there is no evidence 
that ASCAT is “unstable” (as was stated in Leroux et al.) or two 
to three times more biased than the SMOS soil moisture data. 

Furthermore, when the R and SEE values are considered 
the skill values for all data sets (including the AMSR-E and 
ECMWF products considered only by Leroux et al.) are similar 
(to within the statistical uncertainties and within the framework 
of this comparison). The statement in Leroux et al. that SMOS 
“represents an improvement by a factor of 2-3 compared with 
the other products” is therefore valid only for the bias and only 
applies relative to AMSR-E and the ECWMF data in the now 
obsolete version investigated in [1]. 

More generally, the comparison of our results with those 
obtained by Leroux et al. illustrates how different choices in the 
selection, processing, and interpretation of the data may lead to 
different conclusions and rankings of soil moisture data sets. 
Despite the vast body of literature that deals with the validation 
of satellite soil moisture data, there is still a need for scientific 
studies that investigate different assumptions and choices made 
in the validation process (selection of study sites, screening and 
pre-processing of data sets, sample size, use of different error 
metrics, etc.). Additionally, future evaluation efforts for re- 


motely sensed soil moisture products could benefit from novel 
statistical approaches such as the triple collocation method [17], 
[18], the R-metric [19] or the use of alternative error metrics 
[12], [20]. However, we must recognize that the assessment of 
products also needs to consider the intended application and 
that different users may value a particular product and metric 
more than another. Thus, providing alternative metrics is a 
desirable goal. 
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